18 Jointly Normal Distribution

Suppose ${\displaystyle Z\sim N(0,1)}$. Then, for fixed constants ${\displaystyle a,\mu \in \mathbb{R}}$,$$X=aZ+\mu \sim N(\mu ,a^2).$$

where ${\displaystyle Z_1,\cdots ,Z_n\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}}{\sim }N(0,1),A\in {\mathbb{R}}^{n\times m},({\mu }_1,\cdots ,{\mu }_n{)}^{\mathrm{T}}\in {\mathbb{R}}^{n\times 1}}$.

1. Each ${\displaystyle X_i}$ is a normal RV.
2. Being jointly-normal is more strict than just being normal marginally. Because each ${\displaystyle (X_i-{\mu }_i)}$ has to be a linear combination of the same set of iid ${\displaystyle N(0,1)}$ RVs.

Does ${\displaystyle Z_1,Z_2\sim N(0,1)}$ leads to ${\displaystyle Z_1+Z_2}$ normal? No! Independence is important.

Covariance Matrix: random vector ${\displaystyle \overrightarrow{Y}=[Y_1,\cdots ,Y_n{]}^{\mathrm{T}}}$. $$\begin{array}{rl}\mathrm{Cov}(\overrightarrow{Y})& =\mathbb{E}[\overrightarrow{Y}\cdot {\overrightarrow{Y}}^{\mathrm{T}}]-\mathbb{E}[\overrightarrow{Y}]\cdot \mathbb{E}[\overrightarrow{Y}{]}^{\mathrm{T}},\\ [\mathrm{Cov}(\overrightarrow{Y}){]}_{ij}& =\mathbb{E}[Y_i{Y}_j]-\mathbb{E}[Y_i]\mathbb{E}[Y_j]=\mathrm{Cov}(Y_i,Y_j).\end{array}$$

Note:

1. ${\displaystyle \mathrm{Cov}(\overrightarrow{Y})}$ is a symmetric matrix.
2. ${\displaystyle \mathrm{Cov}(A\overrightarrow{Y}+\overrightarrow{b})=A\mathrm{Cov}(\overrightarrow{Y})A^{\mathrm{T}}}$.

Let ${\displaystyle Z_1,\cdots ,Z_n\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}}{\sim }N(0,1)}$. ${\displaystyle \overrightarrow{Z}=(Z_1,\cdots ,Z_n{)}^{\mathrm{T}}}$.
Let ${\displaystyle {\overrightarrow{X}}_{n\times 1}=A_{n\times n}{\overrightarrow{Z}}_{n\times 1}+\overrightarrow{\mu }}$, and ${\displaystyle A}$ is invertible. ${\displaystyle A,\mu }$ fixed. Then ${\displaystyle \mathbb{E}[\overrightarrow{X}]=\overrightarrow{\mu },\mathrm{Cov}(\overrightarrow{X})=A\mathrm{Cov}(Z)A^{\mathrm{T}}=A{A}^{\mathrm{T}}=\mathrm{\Sigma }}$. Then$$\begin{array}{rl}{f}_{\overrightarrow{X}}(\overrightarrow{x})& =f_{\overrightarrow{Z}}(A^{-1}(\overrightarrow{x}-\overrightarrow{\mu }))|det(A^{-1})|\\ & ={\left(\frac{1}{\sqrt{2\pi }}\right)}^n{\mathrm{e}}^{-[A^{-1}(\overrightarrow{x}-\overrightarrow{\mu }){]}^{\mathrm{T}}[A^{-1}(\overrightarrow{x}-\overrightarrow{\mu })]}|det(A^{-1})|\\ & ={\left(\frac{1}{\sqrt{2\pi }}\right)}^n{\mathrm{e}}^{-(\overrightarrow{x}-\overrightarrow{\mu }{)}^{\mathrm{T}}(A{A}^{\mathrm{T}}{)}^{-1}(\overrightarrow{x}-\overrightarrow{\mu })}|det(A^{-1})|\end{array}$$Note that

So$$f_{\overrightarrow{X}}(\overrightarrow{x})=\frac{1}{(2\pi {)}^{\frac{n}{2}}\sqrt{det(\mathrm{\Sigma })}}{\mathrm{e}}^{-(\overrightarrow{x}-\overrightarrow{\mu }{)}^{\mathrm{T}}{\mathrm{\Sigma }}^{-1}(\overrightarrow{x}-\overrightarrow{\mu })}.$$

We have ${\displaystyle X_1\perp \perp X_2⟺\rho =0}$. (not true for general RVs)

For ${\displaystyle M\in {\mathbb{R}}^{n\times m},\mathrm{rank}(M)\le min(n,m)\Rightarrow M{M}^{\mathrm{T}}}$ is invertible ${\displaystyle \Rightarrow \mathrm{rank}(M{M}^{\mathrm{T}})=n\Rightarrow m\ge n}$.

Suppose ${\displaystyle \overrightarrow{X}\sim N_n(\overrightarrow{\mu },\mathrm{\Sigma })}$, with ${\displaystyle \mathrm{\Sigma }}$ invertible.$$\overrightarrow{X}=\left[\begin{array}{c}{\overrightarrow{X}}_a\\ {\overrightarrow{X}}_b\end{array}\right],\overrightarrow{\mu }=\left[\begin{array}{c}{\overrightarrow{\mu }}_a\\ {\overrightarrow{\mu }}_b\end{array}\right],\mathrm{\Sigma }=\left[\begin{array}{cc}{\mathrm{\Sigma }}_{aa}& {\mathrm{\Sigma }}_{ab}\\ {\mathrm{\Sigma }}_{ba}& {\mathrm{\Sigma }}_{bb}\end{array}\right].$$
Precision matrix ${\displaystyle \mathrm{\Lambda }=\left[\begin{array}{cc}{\mathrm{\Lambda }}_{aa}& {\mathrm{\Lambda }}_{ab}\\ {\mathrm{\Lambda }}_{ba}& {\mathrm{\Lambda }}_{bb}\end{array}\right]={\mathrm{\Sigma }}^{-1}}$.

A useful result:

where ${\displaystyle S=(A-B{D}^{-1}C)}$ is the Schur complement of ${\displaystyle D}$ in ${\displaystyle M}$.
Marginal distribution: $${\overrightarrow{X}}_a\sim N_k({\overrightarrow{\mu }}_a,{\mathrm{\Sigma }}_{aa}),{\overrightarrow{X}}_b\sim N_{n-k}({\overrightarrow{\mu }}_b,{\mathrm{\Sigma }}_{bb}).$$
Conditional distribution:
${\displaystyle ({\overrightarrow{X}}_a|{\overrightarrow{X}}_b={\overrightarrow{x}}_b)\sim N_k({\overrightarrow{\mu }}_{a|b},{\mathrm{\Sigma }}_{a|b})}$, where $$\begin{array}{rl}& {\overrightarrow{\mu }}_{a|b}={\overrightarrow{\mu }}_a+{\mathrm{\Sigma }}_{ab}{\mathrm{\Sigma }}_{bb}^{-1}({\overrightarrow{x}}_b-{\overrightarrow{\mu }}_b)={\overrightarrow{\mu }}_a-{\mathrm{\Lambda }}_{aa}^{-1}{\mathrm{\Lambda }}_{ab}({\overrightarrow{x}}_b-{\overrightarrow{\mu }}_b),\\ & {\mathrm{\Sigma }}_{a|b}={\mathrm{\Sigma }}_{aa}-{\mathrm{\Sigma }}_{ab}{\mathrm{\Sigma }}_{bb}^{-1}{\mathrm{\Sigma }}_{ba}={\mathrm{\Sigma }}_{aa}^{-1}.\end{array}$$
Independence:
For ${\displaystyle i,j=1,\cdots ,n}$: ${\displaystyle X_i\perp \perp X_j⟺{\mathrm{\Sigma }}_{ij}=0}$.
Affine transformation:
${\displaystyle \overrightarrow{Y}=B\overrightarrow{X}+\overrightarrow{\nu }=B(A\overrightarrow{Z}+\overrightarrow{\mu })+\overrightarrow{\nu }=BA\overrightarrow{Z}+(B\overrightarrow{\mu }+\overrightarrow{\nu })}$, ${\displaystyle B\in {\mathbb{R}}^{n\times n}}$ invertible, then$$\overrightarrow{Y}\sim N_n(B\overrightarrow{\mu }+\overrightarrow{\nu },B\mathrm{\Sigma }{B}^{\mathrm{T}}).$$